Skip to content

gh-141444: Replace dead URL in urllib.robotparser example#144443

Merged
serhiy-storchaka merged 1 commit intopython:mainfrom
kovan:gh-141444-robotparser-url
Feb 4, 2026
Merged

gh-141444: Replace dead URL in urllib.robotparser example#144443
serhiy-storchaka merged 1 commit intopython:mainfrom
kovan:gh-141444-robotparser-url

Conversation

@kovan
Copy link
Contributor

@kovan kovan commented Feb 3, 2026 •
edited by github-actions bot
Loading

Summary

  • Replace defunct musi-cal.com URL with python.org in the RobotFileParser documentation example
  • Simplify the example to show basic can_fetch() usage since the original output values were specific to musi-cal.com's robots.txt

Closes #141444

Test plan

  • Verified python.org/robots.txt is accessible and returns valid robots.txt
  • make check in Doc/ directory passed

🤖 Generated with Claude Code

📚 Documentation preview 📚: https://cpython-previews--144443.org.readthedocs.build/

@kovan kovan requested a review from berkerpeksag as a code owner February 3, 2026 19:53
@bedevere-app bedevere-app bot mentioned this pull request Feb 3, 2026
Open
@bedevere-app bedevere-app bot added docs Documentation in the Doc dir skip news labels Feb 3, 2026
@github-project-automation github-project-automation bot moved this to Todo in Docs PRs Feb 3, 2026
@picnixz picnixz closed this Feb 3, 2026
@picnixz
Copy link
Member

picnixz commented Feb 3, 2026

We do not accept changes generated solely by AI. Please read https://devguide.python.org/getting-started/generative-ai/.

@kovan
Copy link
Contributor Author

kovan commented Feb 3, 2026

But the contribution is useful and/or productive; I don't think you are reading the document right. Please re-read.

@picnixz
Copy link
Member

picnixz commented Feb 3, 2026

I do think we need to update this but we don't accept AI-generated PRs.

@picnixz
Copy link
Member

picnixz commented Feb 3, 2026

In addition, the example is now smaller which isn't really good. I would also prefer that the comments in the #141461 are taken into account.

@kovan
Copy link
Contributor Author

kovan commented Feb 3, 2026

The document is quite clear: as long as the contributions are useful or productive, it is allowed, or, rather, left to the mantainers to decide.
Maybe it could be rephrased better to avoid this kind of confussions, but I will repeat: I think you are not reading it correctly.

@picnixz
Copy link
Member

picnixz commented Feb 3, 2026

And as a maintainer, I've decided that (1) removing most of the example is not useful (2) not addressing previous PR's comment is not useful either. So, you need to address them. Simply generating this PR without acknowledging what has already been discussed in #141461 is not acceptable.

@kovan
Copy link
Contributor Author

kovan commented Feb 3, 2026 •
edited
Loading

OK, now we are in talking terms. Please reopen the PR, so I can work on it; and the same for the other PR you closed.

@picnixz
Copy link
Member

picnixz commented Feb 3, 2026

The other PR is not useful. I decided not to update the documentation.

@picnixz picnixz reopened this Feb 3, 2026
@StanFromIreland
Copy link
Member

StanFromIreland commented Feb 3, 2026

As previously discussed, it would be better to find a non-empty example.

picnixz
picnixz requested changes Feb 3, 2026
View reviewed changes
Copy link
Member

@picnixz picnixz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use a non-trivial example.

@bedevere-app
Copy link

bedevere-app bot commented Feb 3, 2026

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

And if you don't make the requested changes, you will be put in the comfy chair!

@StanFromIreland StanFromIreland mentioned this pull request Feb 3, 2026
Merged
@StanFromIreland
Copy link
Member

StanFromIreland commented Feb 3, 2026

We could do something like python/pythontestdotnet#7?

@kovan @claude
pythongh-141444: Replace dead URL in urllib.robotparser example
da1cb06 
Replace the defunct musi-cal.com URL with pythontest.net in the
RobotFileParser documentation example. This uses CPython-controlled
infrastructure and provides non-None values for request_rate and
crawl_delay, addressing feedback from previous PR python#141461.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@kovan kovan force-pushed the gh-141444-robotparser-url branch from cc57fff to da1cb06 Compare February 3, 2026 21:40
@kovan
Copy link
Contributor Author

kovan commented Feb 3, 2026

I have made the requested changes; please review again

@kovan kovan closed this Feb 3, 2026
@bedevere-app
Copy link

bedevere-app bot commented Feb 3, 2026

Thanks for making the requested changes!

@picnixz: please review the changes made to this pull request.

@bedevere-app bedevere-app bot requested a review from picnixz February 3, 2026 21:43
@kovan kovan reopened this Feb 3, 2026
picnixz
picnixz approved these changes Feb 4, 2026
View reviewed changes
Copy link
Member

@picnixz picnixz left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those changes look better.

@serhiy-storchaka serhiy-storchaka merged commit 34e5a63 into python:main Feb 4, 2026
55 of 62 checks passed
@github-project-automation github-project-automation bot moved this from Todo to Done in Docs PRs Feb 4, 2026
@serhiy-storchaka serhiy-storchaka added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Feb 4, 2026
@miss-islington-app
Copy link

miss-islington-app bot commented Feb 4, 2026

Thanks @kovan for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

@miss-islington-app
Copy link

miss-islington-app bot commented Feb 4, 2026

Thanks @kovan for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Feb 4, 2026
@kovan @claude @miss-islington
pythongh-141444: Replace dead URL in urllib.robotparser example (pyth…
8937e76 
…onGH-144443)

(cherry picked from commit 34e5a63)

Co-authored-by: kovan <217326+kovan@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
@bedevere-app
Copy link

bedevere-app bot commented Feb 4, 2026

GH-144464 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label Feb 4, 2026
@bedevere-app
Copy link

bedevere-app bot commented Feb 4, 2026

GH-144465 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Feb 4, 2026
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Feb 4, 2026
@kovan @claude @miss-islington
pythongh-141444: Replace dead URL in urllib.robotparser example (pyth…
56e1be8 
…onGH-144443)

(cherry picked from commit 34e5a63)

Co-authored-by: kovan <217326+kovan@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
hugovk pushed a commit that referenced this pull request Feb 4, 2026
@miss-islington @kovan @claude
[3.13] gh-141444: Replace dead URL in urllib.robotparser example (GH-…
e5d8b78 
…144443) (#144465)

Co-authored-by: kovan <217326+kovan@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
hugovk pushed a commit that referenced this pull request Feb 4, 2026
@miss-islington @kovan @claude
[3.14] gh-141444: Replace dead URL in urllib.robotparser example (GH-…
fddd858 
…144443) (#144464)

Co-authored-by: kovan <217326+kovan@users.noreply.github.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead approved these changes

@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes

@picnixz picnixz picnixz approved these changes

@StanFromIreland StanFromIreland StanFromIreland approved these changes

@berkerpeksag berkerpeksag Awaiting requested review from berkerpeksag berkerpeksag is a code owner

Assignees

No one assigned

Labels

docs Documentation in the Doc dir skip news

Projects

Docs PRs
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Dead example URL in urlib.robotparser documentation

5 participants

@kovan @picnixz @StanFromIreland @gpshead @serhiy-storchaka